Papers with natural language understanding models
Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent efforts to create challenge benchmarks that test the abilities of natural language understanding models have largely depended on human annotations. |
| Approach: | They propose a framework for automatic reasoning-oriented perturbation of question-answer pairs that decomposes a question into reasoning steps required to answer it and generates new question-anchor pairs. |
| Outcome: | The proposed framework generates evaluation sets for reading comprehension benchmarks and generates examples without human intervention. |
Overcoming Poor Word Embeddings with Word Definitions (2021.starsem-1)
Copied to clipboard
| Challenge: | Modern natural language understanding models depend on pretrained word embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. |
| Approach: | They propose a method to improve a model's ability to learn to use definitions in natural text to overcome this handicap. |
| Outcome: | The proposed model learns to use definitions in natural text to overcome this handicap. |
Reducing cohort bias in natural language understanding systems with targeted self-training scheme (2023.acl-industry)
Copied to clipboard
| Challenge: | In deep learning models, it is hard to capture all the variations of the language that different users can use. |
| Approach: | They propose a framework that uses four active learning strategies to identify important samples coming from new users and a self training phase where a teacher model is trained from the first phase to expand the training data with relevant cohort utterances. |
| Outcome: | The proposed framework reduces the bias related to new customers in a digital voice assistant system by using two phases: a fixing phase and a self training phase. |
Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2021.findings-acl)
Copied to clipboard
| Challenge: | In July 2019, RoBERTa was the first to surpass a human baseline on GLUE . since then, 13 more methods have outperformed humans on the GLu leaderboard . |
| Approach: | They found that 75% to 90% of correct predictions of BERT-based classifiers remain constant after input words are randomly shuffled. |
| Outcome: | The proposed model outperforms humans on GLUE and SQuAD 2.0. |
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Recent advances on neural approaches to natural language processing have triggered a resurgent interest on building intelligent open-domain chatbots. |
| Approach: | They propose a dialoguE COntradiction DEtection task and a conversational dataset . they show that their best contradiction detection model correlates well with human judgments . |
| Outcome: | The proposed model is more robust and generalizes well on analysis and out-of-distribution dialogues than standard (unstructured) Transformer models that explicitly hinge on utterance structures are more robust, the study shows . |
Robustness Testing of Language Understanding in Task-Oriented Dialog (2021.acl-long)
Copied to clipboard
Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, Minlie Huang
| Challenge: | a lack of systematic studies on the robustness of language understanding models in task-oriented dialog systems is limiting . authors propose a model-agnostic toolkit LAUG to approximate natural language perturbations . |
| Approach: | They propose a model-agnostic toolkit LAUG to approximate natural language perturbations for testing the robustness of language understanding models in task-oriented dialog systems. |
| Outcome: | The proposed toolkit reveals critical robustness issues in state-of-the-art models. |
Local Structure Matters Most: Perturbation Study in NLU (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows that neural models are insensitive to word-order perturbations, but other studies suggest that models learn some abstract notion of syntax. |
| Approach: | They develop order-altering perturbations on the order of words, subwords, and characters to analyze their effect on neural models’ performance on language understanding tasks. |
| Outcome: | The proposed models are insensitive to word-order perturbations while the local ordering remains relatively unperturbed. |
Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders (2022.emnlp-main)
Copied to clipboard
Ivan Vulić, Iñigo Casanueva, Georgios Spithourakis, Avishek Mondal, Tsung-Hsien Wen, Paweł Budzianowski
| Challenge: | a novel framework for task-oriented dialog ToD systems is proposed . a task of recognizing the user's intent or goal from their utterance is a crucial component of any TOD system. |
| Approach: | They propose to transform general-purpose sentences into task-specialized SEs by contrastive fine-tuning on annotated multi-label data. |
| Outcome: | The proposed framework yields effective mID models with large gains over non-specialized models across a spectrum of different m ID datasets. |
Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors (2023.emnlp-main)
Copied to clipboard
| Challenge: | proposed method combines back transcription with fine-grained technique for categorizing speech recognition errors . proposed method relies on the use of synthesized speech in place of audio recording . |
| Approach: | They propose a method for investigating the impact of speech recognition errors on NLU models . they use a back transcription procedure and a fine-grained technique for categorizing errors . |
| Outcome: | The proposed method relies on synthesized speech in place of audio recording to evaluate the model. |
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective (2023.findings-acl)
Copied to clipboard
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang
| Challenge: | Pre-trained language models (PLMs) have improved generalization performance but the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks. |
| Approach: | They propose to create a benchmark for evaluating out-of-distribution (OOD) generalization in NLP models. |
| Outcome: | The proposed benchmarks highlight the importance of OOD robustness and provide insights on how to measure it and improve it. |